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Iritroduct ibri 

This paper was prisparisd" under>the guidance of the title of f 
this session: Perspectives on the Documentation of School Improve- 
ment Efforts; Philosophical riMethodoiogicai , and Practical Cohside- ^ 
rations. In the past decade, I have sponsored evaluation work 
and administered federal research, demonst r at,ion and entitlement 
efforts in education. This paper will draw heavily on that experi- 
ence but, hopefully, be within the bounds of the* guidance . 

In reviewing the background of those participating in these 

paper^ writing sessions, I appear to be the only entitlement program 

administrator presenting / therefore I hope to make this a somewhat 

personal set of observations oh the acquisition and ujfellization of 

* J 

"facts'* by federal administrators. The interpretation and generali-r 

zatibris which might come from these observations are left, in part> 
to the reader* and the excellent discussants. 

Four groups of observations comprise this paper. The first 
part discusses - the role of belief and knowledge in program adminis- 
tration. The second describes the federal program evaluation — what ^ 
a federal program evaluation is beginning to appear to be as we enter 
the eighties. The third se^ of observations deals briefly with the 
dissemination of diffusion of evaluation work, and the fourth pro- 
vides some observations on the purchase of evaluation combohehts. 

National Pr-ob lems and Feder -aJ^lssues y 

Our purpose in these sessions is to offer adi^ice to people 
planning and implementing efforts to document schoW imprdvemeht 
efforts. The federal sector has channeled large humbles of dollars 
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into fairjy visible efforts i However, I feel it is important to 
begin this paper by raising the distinction between natiJrfal problems 
and federal issues. ■ , . 

Even with billions of dollars in the £f»derai budget dedicated 
to educational endeavors , pebplj^ cont inue^ to ask that the federal 
role ,b^ clarified. There 5rs a cbhtihuihg uneasiness about the coher- 
ence of the federal efforts and a continuing uneasiness about the 
intrusiveness of the^^ederal sector, into state arid local matters. 

There are a n Jmbe r oINl p r obi ems which face classroom teachers , 
buildings and district devei admirtist r ator s r and state level adminis- 
trators. More critically, the probiems faced by an individual at 
any level within a state is often the same as pPobieras faced by a' 
counterpart in other states. For example, when the metric conversion 
began to be a real pbssibilityr curriculum content and classroom 
mater ials wer ^ very 1 imi ted . Cbrigress dec ided^ because the problem 
was a frequent ef'ent, a national problem, a^riatiorially funded 
effort should stimulate the production of curriculum and material. 
The Congress did not se8 this as a federal curriculum — needless 
to say. 

, » — 

. _ _ _ • ^ 

National problems are qaite cifli^iy those problems which (1) 
appear with some frequency and (2) a^S^r to be distributed quite 
widely amorig the states. The field of education confronts such 

problems. For example, childreri iri low incomes schools as 3 group, 

_____ • ^ 

score more poorly on standardized achievement tests than children • 
from middle class neighborhoods. Even with supplemental funds 
sucFS as Title I of the Elementary and S«fcondary Act flowing to 



- 2 - 



ERIC 



these schools r the problem persists. This gave rise to legis- 
lation to authorize federally approved demons tratidri activities 
to assuxe that successful programs were developed and given visi- 
bility. In this example, the perceived problem was the money 
available for quality programming in individual schools. The Con- 
gres3 decided to intervene, but not with a federal curriculum. 

For the sake of argument, I would tike to suggest that many 
edjtica tibnal problems are " unha tur al " ' f eder al issues. I used the 
terra "unnatural" because the iristrumeri tali ty of the federal govern 
merit iri education is riot designed to assist childreri directly 
(^ith the exception of the defense schools) . The national contact 
with children is either through the state gdverrimerit arid the 
local" educatiort agency o.r , ort occasion, ^S^passirig the state arid 
going directly to the local- education association. The Education 
for ftii Handicapped Act (P.L. 94-142) provides an illustration . 
This /^ct creates numerous ^changes in the way irs^which the schools 
i/riteract with harid icapped" childreri and their parerits. However , 
the state is the agericy with whom the federal furidirig agericy deals 
Iri a^temptirig to mdriitbr compliance with this Act federal staff 
visit schools and classrooms and families withiri states. However, 
the reason for these visits is to assure that (1) policies which 
had previously been reviewed at the state capitol were dissemi- 
nated to and 'Under stood by local education agencies and (2) 
mbriitoring by the state was sufficient to ^assure that violations 
would be fourid;and Corrected. Federal teams fi-nd discrete vio- 
latibris oi the federal statute rather freqderitly. These violations 
are referred to the state for Ebllbw^up. The violations of 
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more direct concern for •federal officials are those which suggested 

: ^ 

that systematic problems were occurring; problems which might 
be related to inadequate monitoring by the state officials or 
inadequately^ or unwiliihgly dissemination and implementation 
of state policies which hav^ been Redrafted to be in cdnfbrmarice 
with EHA. ^ 

As concerned human beings, we wanted the chi/dren to get 
the best quality prb^rammirig possible. As federal bureaucrats, 
we did not deal with these events^ except to the extent reoccur- 
ring problem^ may have represented policy inconsistencies between 
federal and state statutes or regulations^ or may have r\epresented 
a weak strategy of local school monitoring by the state. In the 
area of the handicapped, the federal issue was the extent to which 
the state was actively pursuing the goal -of a "free, appropriate 
public education" for those handicapped students. The reason for 
looking at the children's program was to assess the state's actions. 

The Office of Special Education in the Education Department 
also has demonstration programs. The excellent people ruhhihg 
these programs are primarily interested in prbducin^high quality 
experiences for children who happen to have handicaps. As a 
federal administrator, I had the following^ goals for the program: 
(1) Quality. Evaluation was essential By the project developers, 
we could'^bt affort to have the credibility of these demonstra- 
tion challeng^ed. The purpose was. to display excellence. 1/ 

(.2) Distribution. .The projects had to be widely distributed both 

^ _ >_ _' 

among the states (We wou^ld send a team into states to dram up 
interest if one- was without a project.); among urban and rutai 
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areas; among distinctive subgroups of the American "pdpulationi 

(3) Recruitment. The field which provides education services 
to young children was judged to be tod small arid too thin. One 

of the major purposes of demonstrations is to create arid riurture 

. - - ■ . 

, performing fields. fts a result, the number of persbnriel trairied, 

the number of^pubiicat ions , the location of publications, arid the 

presentations in national forums were carefully monitored, 

(4) Diffusion. The projects served every handicapped condition. 
All were supported with a mix of federal and local or state funds. 
(A coridition of the grant was the strong indication' that local 
funding would assume the cost of the project at the end of three 
years.) Projects were lobated iri very different settings frT order 
to have examples which be a challenge to ariy skeptic. As a result, 
there were devastating arguments against people who said: "It ^ari- 
hot be done..' These children will not benefit from an education." 

.As a result > the project became visitor centers — and the number"^ 
of visitors were monitored in sites. They distributed information — 
and the numbers df^materials developed were monitored. A subset 
of the projects were giveri furids for travel and training at the 
end of three years but no more project furids. The projects had 
to be alive on local or state funds. 

The demonstrations were designed to meet a widely distributed 
problem in this country. Programs for children ^ge 0-6 were few 
in number, poorly distributed, and the personnel had neither time 
ribr money to monitor CSeir effectiveness much less study and 
write about w^at they were doing. 

- 5 - 
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The Congress assumed that provision of services was ^ good 
idea and that such, prpgrams would be effective. _2/ What was desired 
was greater distribution of services. This required people 'and 
cbitiinuriication . The federal issue was to attracjt people to the 
cause and help networks d.evelop across state boundries. I 
pur pose that these issues are reasonably "natural" for the f eder al 
establishment. The programming that goes on within the site' is a 
l^ss '•natural^ agenda for the federal establishment. . . 

The perspective this line of argument is developing is that' 
the primary federal -issues af intent when dbcumehtihg, these school 

iiTiprbvemerit efforts are those havin<^ to do with the level of the 

_ _ _ 

human resources, actively engaged in assisting handicapped children 

__ _ s 

and the characteristics of the network Which is being developed 
to diffuse this program. A person working in a school site or 
at the state department of education would have a, different per- . 
spective an^ therefore a different set of focal variables. The 
nature arid quality of the. specific cur r icuium intervention 'would 
be a "natural" interest. ' 

6* - ^ 

B elief and ^^ nowledge " 

A perspective on the documentation of school imprbvemeat 
efforts changes when moving from the administration for federal 
demonstration prograiis to the administration of federal entitle- 
ment progams. By defiriitiori administrators and pol icy maker s "^te 
constantly performing. A program exists. Their job to run 
and modify the program. As a result, administrators and policy 
makors run on what they have in the way of l:^elief and knowledge. 



since i feei the importance of belief in the operation: of this 
couritry/s business, is underrated^ the purpose bf-this section is 
to discuss the terms ''belief" arid " krio wledge" ^ and emphasize the 
sigriificarit role of belief iri'the admiriistr atibri of programs. 

Bertrand .R^^sel'l defined bel ief and knowledge in a useful 
chapter in Human Knowled ge , y/ Belief was defined by a number . 
of descriptions. An example ^f a belief appears when a person 
iays, "The child is handicapped, sh^. will perform poorly'" A 
belief is also displayed by the |)erson who says, ''the school has 
unstructured classrooms. The place is going to be chaotic." 
Belief then is the state or cbriditibn bf a per son ' s^tHiri king . 
The belief has ari exterrial refererice. The people in the examples 
above encbunter situations arid expect a giveri ev^erit because* 
of a belief. ^ This also illustrates the relationship of facts 
to beliefs. Russell notes that truth is a property of beliefs.^ 
The truth of a belief depends on having some intelligible ej^peri-^ 

ence (fact) behind it. Whether you are administering federal 

• _ _ 

programs or guiding a family, there are abundant facts for* all 
sbrts bf beliefs. True beliefs are plentiful", conflicting and 
often difficult tb fbrget. • , 

Iri cbritrast, knowledge is a subset of bel.lefs. Iri the area 
of education, for example, there are nlUTieifous examples of belief S| 
which are not knowledge. The belief that highly structured set- 
tings .are important for child learning is supported by intelligi- 
ble experience; by numerous " facts . However, the belief that 
undefined space is important for child learning is similarly, 
supported by experience or facts. Sbme larger, more encompasing 

■'. ■ 7 - ^ ■ ■ • 
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concept is proba^Dly the generic basis for both of these beiiefsi 



Simiiariy, many believe that, tension is destructive to perfbrmarice 

Others believe th^t tension is necessary for performance. These 

beliefs are both ' suppor ted by a number, of facts. The evidence . 

that per fbrm^nce has been^fbund to have an inver ted "U" shSpe 

furictibri as tefisibn is increased according to a linear fuhctibn 

allows us to have knowledge which encompasses the two apparently 

conflicting beliefs. - 

Russell defines knowledge as having a somewhat. higher 

standard to met than Relief. .-4/ 

What char acter in addition to truth must a belief 
^ have in' order to count as knbwledgfe? The plain 
inan would say that' there must be sbuhd evidence 
to support the belief. As a matter bf cbmmbn 
sense this is right in mbst of the cases_iri which 
dbubt arises in pract^ice, but if intended as a 
complete account of the matter it is very inade- 
quate. "Evidence" cbnsist^^ on the one hand, of ' 
certain matters of fact that are accepted as 
undebatable, and/ on the other hand of certain 
principles by means of which inferences are 
' drawn from the mattelrs of fact. It is obvious 
that this process is unsatisfactory unless we 
know the matters of ^act and the pr inciples^o f 
inference not merely^by means of evidence,, for 
o therwi se we become involved inaviciouscircie 
or an endless regress. We must therefore con- 
centrate our attention bh the matters of fact 
and the principles of inference , We may then ' 
say that what is known cbrisists, first', bf 

certain matters o f f ac t _and cer ta in !pr inc ipl e-s ' 
of inference^ neither bf which stands in need 
of extraneous evidence^ arid sec(Xndly> of all' 
that cari be ascertained b-y applying the pririci- - 
pies of irifererice to matters of fact. 



No administrator waits until knowledge is available . As aNresu^lt , 
'^administration by^ belief" and "policy making by belief "^s 3 * 
very frequent and necessary event. The point,;of this description 
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is to acknowledge the irngbrtahce for thege two types of ''kribwirig" 
to administrators and to suggest that evaluatbrs can p.lay an 



important roie in the de^velopment ctf—feroth types of human knowledge. 
There are a numbe.r of x:on tr ibut ions made by the evaluation cominunity 
to test the belief systems underlying administration and policy 
making, and I feel that this is important. For example, executive 
branch personnel often act on beliefs about the goals for programs 
•which are not the same as those held by the legislative branch. ' ; 
The most complex condition is found when the executive, the legis- 
lative branch ^rtd the public all have di f £erent bel ief s about 
the goals. Methods have been developed to get more achieved > * 
S^t a minimum, awareness among different parties, or po^ssibly even 
congruence among, different parties. Another exam^)le are -the 
techniques which have been developed to generate new beliefs. 
The Department of Health, Education and Welfare, under the ^leader- 

ship of Joseph t^hble^ attempted to^ systematically collect facts 

____ • _ _\ 

which would lead to hew' and bbmpeting beliefs as well as finding 

additional facts to support bid beliefs. One example of what 
were called "service delivery assessments" was a study which 
looked at the services the Unacted States was providing the 
Indochinese refugees. 5/ in this study, the belief that refugees 
are, homogenou? group was seriously c^iiengedT The study identi- 
fied, for the purpose of service delivery^ quite distinctive 
groups, of refugees. The first group was composed of active pro- 
fessiorials who needed tb have infornvatioh, access to licenses and 
a limited amount of physica:l support tb disappear into the American 
system. A second group had little going for them. They were 
. • • 9 - 
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unskilled. They had majo^ physical and/or menifeai handicaps. They 
were very limited par ticipants in their own couhtr ies before 
becoming refugees in the United States.^ While the first group 
often had m;al til ihgual individuals, the individuals in tliis group 
were often illiterate in their home . language . In between you 
had other groups. While these facts may" have been related -^o 
< the beliefs of those administrators close to the program, they 
have the potential of giv-ing a department secretary or a congres- 
sional bversite committe<& a basis for quest ion^ing, the appropriate- 
ness of service delivery strategies. For ;the administrator, it 
gives a chance to test the "common sense" and for the researcher / 

the service delivery assessment gives some clues to what is worth 

_■ - . ^ 

investigating and how to define variables. 



•Summary ^ 

The position beting conveyed in this section is that belief 
is a very important basis for action. Techniques which improve 
the evidential basis for beliefs and result in the strengthening 
or the replacement of previously, held beliefs, in turn provide 
the basis for improving services. if evaluation personnel want 
to be primarily involved witfi knowledge production/ they will 
be bhiy partially involved with policy mafjing and admiriistratibh - 
Knowledge production recjuires the cor rect ^questijDn or issue and 
then a lot of time, and skill. The correct question or issue can 
probably only be developed after a clash develops between competing 
arid factually supported beliefs. . * 
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Federal Program Evaluation 

The previous section suggested that some things seemed to 
be unlikely candidates for focal var iables- when documehtirig school 
improvement efforts from the federal pecspective. The purpose 
of this section; risky as it is, is to list the questions which 
seem reasonable to address in federal studies or evaluation of 
school improvement efforts. 

in 1977 i asked Dr • Mary J^ennedy to develop the evaluation 
plan for t'he Education for Ail Handicapped Act. Policy makers 
had a large number of questions which needed to be examined; 

questions characterized by an extensive array of topics and topics 

> _ _ 

characterized by changes which would occur over time . The plan 
is remarkable because Dr. Kennedy achieved agreement on the 
questions and agreement that everything could not be studied 
by one method at one point in time. ^/ 

The type of questions which needed to be addressed in this 
study are listed in the following display. In a recent planning 
effort IJ found these questions to be useful in organizing an 
evaluation plan for a study of a Labor Department program; there- 
fore, there may be some fairly standard questions which should be 
addressed from the federal perspective , across programs . However, 
this assertion needs much more testing. 

The experience of developing this study plan causes me to 
greatly respect the following: 

- 11 - 



EKLC 



13 



DISPLAY • 

A Full Scale Federal Program Evaluation Generally Requires: 
I. Identifying the goals of Congress 

A. Who are the intended beneficiaries? 

B. What services are envisioned for these beneficiaries? 

C. What administrative mechanisms did the Congress envision to provide 
services? 

D. What positive impacts were expected? (What negative impacts were 
to be guarded against?) 

Describing the executive branch's program 

A. Who are the beneficiaries receiving services? 

B. What array of services exist? What is the relative frequency of 
services? " \ 

C. What administrative actions has the executive branch taken? What 
administrative mechanisms are in place? 

D. What impacts appear to be covered by or associated with the presence 
of these services? 

III. Providing an analysis and syntheses of the data collected 

A. Do the intended beneficiaries appear to be receiving services they 
might not have been otherwise receiving as a result of this program? 

B. Do the services being received appear to be consistent with those 
envisioned in the Act? 

e. 00 the actions taken by the.executive branch appear to be consistent 
with thos^ expected by the Congress? (E.g., regulations, distribution 
of effort . ) 

D. Are the impacts identified related to theseryices provided and are 
these impacts consistent with the intent of the legislation. 



IV. Providing recommendations 

A. for the law 

B. for the executive branch 



C. for the local administration of cervices and/or federal funds. 
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(1) brokering of the questions; 

The general question or the starting question always has 
a farailar ring: Does it work? is it successfal? All of us have 
grown use to the difficult trarisitibri^q the next questions 
envblved in defining "success" arid "Work". Sometimes we find 
difficulty in defining the "it" that is the intervention. Success 
usually has many faces in a social program. In order to gain the 
necessary political strength to get programs passed through the 
COrigress^ the purposes of any single program are usually fairly 
expansive. If one Member can see an opportunity to meet an 
objective Or iricreasing employment with a program, this^may prb= 
vide a ratiOriale for a vote. If another Member sees an opportunity 
to help rural districts With a pr\gram, the same program, this 7 
may provide another ratiOriale for a vote. As a result, the 'success 
of any program is usually described along several- different 
dimensions. The dimensions can even be conflicting or competing. 
The unlikely event is the program with a sirigle purpose within 
a perspective much les^ a purpose that is commori ambrig perspectives. 
A bargairi rieeds to be struck. \ 

(2) Bargaining for time. 

ft is quite common to hear knbwledgable people argue for more 
time before evaluations are initiated. If we have learned any- 
thing in the 6Gs and 7Qs , it is that time is required for imple- 
menting innovations (which seem to be complex, almost by definition) 
^■XJ^*^^ evaluation of ERA, data was produced for sbne questions. 
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irnitted lately- Other questions needed to wait for investigation. 
The reasons for delay were based oil the three major reasons: 
1. No idea of what was worth being examined^ 
. 2; No technology to examine what was worth examining 
3- No reasonable ex pec tation that what was to be* 
examined was in place. 
But nodeiay was argued for the evaluation. The bargain for time 
meant that som'e questions would be answered immediately. 

(3) Initiating array of studies. 

The program evaluation is hbt^he same as a study or a con- 
tract. The program evaluation is many questions^ and many studies 
using many methodologies. This is the only reasonable conclusion 
from experience with the Education for All Handicapped Act and 
from any more recent experience with the tabor Department program 
mentioned earlier. Gurrentiy, fifteen studies are included in 
the EHA evaluation. Data is collected at one point in time in 
some studies and over several years in others. Case studies 
are used as well as the national survey. Some data cbllectibhs 
are made once, others are made at intervals over a period of years. 
These choices depend on the discrete questions. 

4; Focusing on descriptive data. 

The overwhelming experience in the EHA evaluation was the 
demand that pi^licy makers and the public (and the courts) have 
for descriptive information. The questions related to who is 
but there and v^at services look like seem to be the central x 
interest of federal administrators. This experience has been 

13 - 
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replicated in a s^tudy of schbbi systems Cdrrentiy supported 
by the National Institute of Evaluatibri, 7./ 

p 

Summary 

Whenever we are invited to document Achieved improvement 
efforts in the future, I feel the style of the approach has been 
well documented by the experiences of the last decade. The 
brokering of questions, the use of several studies with different 
methodologies, the bargaining for time and the focusing on 
descriptive data we will illustrate the successful work of Paul 
Hill on the Title,! evaluation and May Kennedy in the evaluation 
of EHA. 

Gathering Data 

Earlier I described the federal program as many studies 
designed to inform on a limited set of questions. This argues 
for a different strategy of sponsorship than is usually found to 
obtain field data. In the course of procuring the Follow Through 
evaluation, one contractor (SRI In ter national ) was asked to perform 
the entire effort, initially. Everything from planning to data 
cbllectibn, from analyzing data to reporting; from disseminating 
information to organizing constituents (both parents arid academics); 
e tc . Th i s ex pe r ience and othe r s c a us 3d me to feel that it was 
physically and intellectually impossible for a single organization 
to organize and execute a major program evaluation. There are many 
reasons. The first regards people. How can any single organization 
capture enough talent to put, lets say, ten senior invest igatTors 
on a jingle contract? In some areas we are fortunate to finH 
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ten senior performers in the United States much less ten senior 
performers in one organization.^ But even if a number df perfbrtnecs 
are ^ found; in one setting a diversity of intellectual cbmmi tmeri ts ^ 

is difficult to find^^^ Even Universities which are supposed to be 

. -- - , ------------ - . - - - ' ^ 

dedicated to diversity of ideas have departments which are known; 

for*a preponderance of advocates for a particular therapy or a 

particular school of economics or a particular pharmacological 

treatment strategy. The same situation exists irt contract 

/ . - - - " _ _ _^ ^ _j 

research/ firms . People who are very successful with particular 

techniques or strategies usually get a lot of business by clients 

needing those techniques. Recruitment becomes focused. Applicants 

who like a certain approach to things apply. ,The result i-s that 

it is usually difficult to find a group dedicated to survey 

techniques, for example, who have an eqi^ally strong group interested 

in rigi|rous case studies. I also suspect that the associated 

but cjc^te different demands which such various groups would place 

on the management structure of an organization might be difficult 

or even unbearable. 

As a result, when I hear that one RFP has been announced to 

evaluate a given program, I feel the policy maker will be poorly 

served. A single contract usjjally means a single contractor (Sub^ 

contracts are usual but are generally used for highly specialized 

____ ) __ 

technical assistance as opposed to creating alternative or competing 
approaches to the overall task). The likeiihooc3 that a single 
source has the human resources available to conduct a major program 
evaluation striked me as unlikely. If a contractor manages to 
corner this huge talent pool, the likelihood that they can be 
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managed, strikes^me as u.nlikely. Hence, I feel that prdgram 
evaiaations are more serious if there are plans to utilize several 
^diffec^ht resoufces, each thosen because they can contribute uniquely 
to Jan aspect of - the program being studied i 

The second, and related, experience which i Have had 
repeatedly is the- evaluation bf programs which is often based on 
obtaining data through a single investigatory strategy. I feel 
very nervous when a study is proposed which used a questionnaire^ 
for example, to answer a large number 'of interesting and important 
questions. 1 have had two recent experiences where the methddb^ 
iogy was selected and then mal^ginal tinkering was performed to 
get this single basic vehici^'e to nibble at additional questions. 
My nervousness is becoming a reflex. Even though it takes more 
time^ I feel that each question of interest for program evaluations 
must be treated as independent events ^nd the bist p6ssible 

design created for the questions individually. If there is 

» 

sufficient design and sample overla|>^tb justify a'^marriage, I 
would rather make that decision as a. result of seeing opportuni- 
ties for collapsing individual studies into a single study rather 

* . V 

than beginning with a single study and adding side interests. The • 
first sympton of this disease is usually the comment: "Since we , 
are going to go to all bf these cities anyway, why don't we just 
add ..." . 

The better strategy is well illustrated by the studies of 
the handicapped act and ESEA Title' I. When the (evaluatibn fot 
the Education for All Handicapped Act was planned. Dr. Kennedy 
divided the study into seven bo^sic qu-estions. 8/ Individual 
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studies were planned to examine facets of each of the study 
questions over a five year period. As of January, 19B1, 
15 studies had been commissioned with 20 contractors. T^iis also 
brings up another point. The federal of ficiais (^nvolved in 
evaluating the Education for All Hand i capped Act , pc'^l^red an 
annual report which was a syntheses of all studies of interest 
to the date of the publication'. 

As a result, the burden nhich is often 
_______ <t _ 

given to the individual report, the individual technical report, 

to serve as both a technical statement as well as a dissemination 

vehicle, was' . It seems clear to me that both purposes 

are poorly served^lf one document is to speak both to the technical 

cdmniuriity and the political community. A similar strategy was 

pursued by Paul Hill in his major study of Title I of the 

Elementary and Secondary Act. Dr. Hill commissioned studies with 

separate pontractors. He was faced with a shorter time frSrne^ 

years. However, the separate studies provided the cumulative 

evidence desired by the Congress in making its single largest 

education decision in the reauthorization of Title I in 1978. 

Dissemination and Diffusion of Evaluation Research 

We pr id e our se Ive s on be ing or i en ted to empi r icai ev id ence 
therefore, I am certain we will all accept the notion of finding 
that our technical rej^orts are not good vehicles for cori:imun icat ing 
to the public or policy maker. Hbwevei^ little discussion exists 
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on a reasonable alternative. The purpose of this section is to 
make one point about this situation. ^ 

As a person who has been a sponsor of evaluation and research 
efforts for nearly a decade, i am no longer concerned by a criti- 
cism that a report has too much technical information or language. 
In factr the opposite would be more disturbing, :^f a study is 
done and results are reported which have too little technical 
information to allow either vicarious. pr actual replication, then 
there should be criticism. The point being that evaluation 
research reports should.be technically competent and technically 
complete statements. The investigator has the professional obli- 
gation to represent the technical strategies employed in a study 

in the most acceptable and understandable fashion. The obligation 

- 

is to deliver a technically understandable statement. Without 

such a statement, there is no starting point for the bureaucrat 
or the politician. 

It seems reasonable that another vechicie needs to be developed 
tor a 1 sseinma t ion and diffusion to other audiences, fact, several 
audiences might require several vehicles. The form and standards^ 
for reporting evaluation research are develbf5ed. Part of the 
emergence of a profession is usually hearlded by the acceptance 
of a series of rules or standards for performance. The content 
of reports is a well debated standard for scientific wo/k. There 
is no reason why th^se standards should be changed. In fact I 
would argue for the opposite; for more enforcement of the standards 
of gdc>d scientific writing. The vehicles for other audiences 
are quite different. In both the evaluation of the Education 
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for All Handicapped ftct arid in the Revaluation of Title I of the 

Elementary^ and Secondary Act, synthesis reports were made from 

* ^ 

the commissioned literature for thp Congress . -The report prepared 



by br • Kennedy went beyond the' commi^Bsioned studies and searched 

for relevant, fugitive" 'studies . The demand for both of these 
synthesis documents went far beyond any demand ever received by 
these managers for the individual investigations; nonetheless > 
the individual investigations were fully available so that inter- 
pretations and statements could be challenged. ' 

' ____ 
It is dbvidusly not clear what makes the best dissemination 

vehicle for each necessary audience for evaluation reports. How- 
ever/ I feel that debating this question is more reasonable that 
attempting to add another reporting barren to the already difficult 
task of clearly representing what was done in a study for the 
technical communities. 

It seems reasonable to assume that most of the questions 
we now have are difficult to inform with a single study. ^ In fact, 
it seems a bit egotistical for an indivic^ual investigator to feel 
that his or her study should have the goal of clarifying all 
aspects of a major policy question. The more reasonable view is 
that the study is part of the camaiativo process of assembling ^ 
evidence. This, in turn, places the burden on a sponsor to come 
up with an evaluation plan^ not a study. The Congress expected 
such a plan in both the Title I work arid the HaridicappJd Act 
evaluatibri- 
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Fugitive LiteratU|-i5 

there is one final note, that I want to add .here. When the 
sponsor is required to assemble studies, the opportunity and jl 
responsibility exists to examine th¥ '•fugitive" literature. 
The review of extant but unsoonsored literature has tangible 
returns. Even if the "news" is negative, i't is better to Icnow^ 
the wdi:k; In the evaluation of EHA both Dr. Kennedy and I hoped 
to offer sntall cont^racts to the academic commanity to. capture' 
and report^ the unspbhsbred literature. X^e were unafele tp^get such 
contracts established befbre leaving but the i^ea is im^ortarit ' L 

for the^next generation of program evaluations and for the qdality 

•. _ _ . ' _ _ ' 

of the package of information prejpared for th policy maker- 



Fdllbw Through 

Our purpose in this session was to give phi Ibsbphical > 

methbdblogical and practical perspectives on the dbcurnentat ion 

of schbc|l effbrts. As a result of the more general statements, 

the conference brgahizers next wanted us to tarn oar attention 

to the large and -innovative Fbllow Through Program and reflect 

- ^ — ' 

on the future of this program. This progran has already accbm- 

plished many goai.s.' This effort: (l) recrui ted - ,.v . of the very 

talent performers from th^ early childhood Held and diverted 

• 

their attention to the K-3 school years; (2) provided visible 
demonstrations of excellent programming throughout the natibh; 
(3) contributed •to the literature on early school assessment; 
and (4) transferted parts of or the entire program to school 

V 

systems throughout thjs' cbUht ry . \^ - ^" 
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The Fbllbv/ Through dilemma is obvious, what is haxt at this 
point in time. Obvibusly> I believe that demons tratidris are 
potentially powerful ve^cles to * assist, the state arid local 
agencies to overcome widely encountered ' school problems.. I 
also beiii9vi9 that program evaluations as similarly powerful 
tools to assist federal poliqy rnakers; fts a practical flatter,. 
Follow Through should be analyzed from the standpoint that 
these perspectives provide quite different opportunities - * 
For example. Follow Through is unique *becau3^ it has 

a comprehensive mandate (health services are authorized as well 
as educational services). The Education for All Handicapped Act 
has a similar mandate. In administering .EHA we pursued health 
services seriously and found out hbw little we knew about the 
trust funds (Title XIX and XX) and how little the schools 
Understood -about how to cooVdinate and use these funds or' 
services. Follow Through may be uniquely suited^ to allow 
study of the nature of health and education service coordination 

for low income children. This is clearly ah area where the 

J _ . _ . _ .. ._ 

federal executive and the. Congress has heed for more "true" 

beliefs and knowledge. Coordination of service:^ is clearly 

a national prdblern. My choice woull be to viev7 each stated ^. 

as idiosyncratic and see the Fotlow Through project(s) within 

the state examining th^e service opportunities for a given 

state service strategy. In pursuing such an issue, the study 

plan woatd need to be developed and numerous evaluation studies 

commissioned. ; 



There are other hatibhal issues that are worth cbhsiderihg. 
For example^ Follpw Through represents a carefulli^ designed set ; 
of intense training efforts. The National; Science Foundation 
extensively documented its science training strategy in the 6bs. 
the Teacher Corps and teacher Centers represent anot^ier federal 
intervention ahd^ a fairly extensive literature accompanies thesi ^- 
efforts. No attempt has been made to similarly document the Follow 

rough ihtervehtion much legs to discuss in a comparative way> 
the federal efforts across these significant efforts. 

Follow Throu<^ could also be refitted as a hew demonstra- 
tion vehicle. The materials developed by the Follow Through Spbhsdrs 
and tihe philosophies espoused by the Sponsors have had a profound 
impact on the Nation's schools. there are few national con- 
ferences of education research or curriculum leaders that do not 
have either the original Follow through Sponsors or the "children** 
presenting. Perhaps it is tine to see the demonstrations opportuni- 
ties for such a program in the later grades; the grades where 
comprehensive of words ahdihumbers receives priority rather than 
the decoding of words and numbers; the grades where soci^-j^ ah'd 
academic behavior extends to wider and more cbmnlex circles. The 
legislative authority may not exist for this redirec-i^ion . HowevGr, 
my hopes for outcomes from more demonstration work at the early 
elementary grades seem marginaV dqgipared to what might be done 
for pur oiliest elementary school children. 

^ \Vhile 1 c?o not see an obvious possibility for a new national 
evaluatibh, Fbllbw Thrbugh has qcncratc<] extensive evaluation 
literature over the past docadc?. Only a part of this litoraturo 
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was federally initiated. Ah eribrmous fugitive literature exists. 
Neither the executive nor the Congress has been veil served by , 
this literature. I feel an effort to bring this literature together 
around major national problem or around major federal issue areas 
shouiS be considered. These should be separate s taterrients . !Teither 
purpose has begun to exhaust this titerat:ure. 

Follow Through *s future is a ques tionmark , On the one 
hand you have a significant program which provide^ opportunities 
to address federal and national issues ^ Oh the other hand^ the 
executive branch attempted to cut the program in the budgets of 
1975/ 76, 77 .and the program was sa'^Cred by communities which, 
understandably, want and can use the extra resources . The; articu-^ . 
lation of the federal issues toward which this prograrn is addressed 
or the national problems toward which the program is ^addre^sed 
needs to be accomplished. if the program serves neither purpose, 
the research and demonstration communities should be graceful 
in acknowledging the situation* But the Follow Throu<3h t^togram 
should hot disappear without a standing ovation to the peopile who 
accomp^lished so much- . ' /C ' 
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